Reproducible Assessment of General Education

Systematic Use of Course-Level Student Data

Dr. Clifton Franklund

October 24, 2017

About Ferris State

  • West Central Michigan
  • Medium Masters University
  • 14,600 students
  • Recently Revised General Education
  • Opportunity to embed assessment

General Education Goal

  • Track and improve program effectiveness
  • Program consists of courses
  • Student preformance is the most valid measure
  • General Education applies to all students
  • Therefore, we plan to measure student performance across all courses in the program

Project Scope

  • In a semester…
  • Over 300 different course codes
  • Touching over 5,000 students
  • Involving over 300 faculty
  • Generating over 50,000 SCH

Project Resources

  • Program coordinator (me, 50% release)
  • General Education Committee (advisory)
  • Budget ($0)
  • Staff (none)
  • TracDat (data repository)

The Assessment Cycle

The Futile Cycle

A Better Model

“Assessment is not a spreadsheet – it is a conversation”
Irmeli Halinen

Reproducible assessment

Sustainable assessment

  • The three keys to our program assessment strategy:
  • Simplify — reduce complexity
  • Standardize — reduce variance
  • Automate — reduce workload
  • Aiming for reproducible processes to increase transparency and encourage engagement

Simplify: Core Competencies

  • Eight competencies:
  • Collaboration, Communication, Culture, Diversity, Natural Sciences, Problem Solving, Quantitative Literacy, Self and Society
  • Each overseen by a faculty subcommittee (8-10 members)
  • Operational definitions
  • Goals (Hallmarks of a Bulldog)
  • Relevance statements

Simplify: Ferris Learning Outcomes

  • Each competency has four learning outcomes
  • Based upon what we want our graduating seniors to look like
  • Derived from LEAP
  • One outcome measured each semester
  • Rolling two-year program cycle

Standardize: Measures

  • Created in order to get more uniform reporting into TracDat
  • 14 different measures are defined
  • Each measure has its own:
  • Scoring system (zero to four point scale)
  • Analysis interpretation (four threshold values)
  • Criterion leveling (expectations by course level)
  • Each measure is present for all FLOs

Standardize: Registering Courses

  • I need to know who is doing what each semester
  • Who is collecting data?
  • What competency is being measured?
  • What outcome is being measured?
  • What measure is being used?
  • All tracked using a Google form

Standarize: Data Collection

  • This is the biggest step forward for us
  • Excel workbooks to collect and summarize student results
  • SCORE, PREPOST, NORM, RUBRIC
  • Common metadata supplied by instructors
  • Enter student names, IDs, and appropriate evaluations
  • A standaridized summary is automatically generated

Standardize: TracDat Assignments

  • Seeking to minimize the need for faculty to log into TracDat
  • Received as an email
  • Copy and paste summary
  • Select a few responses from drop-down menus
  • Add a short reflection
  • Attach the workbook and submit

Automate: TracDat Overview

  • Allow quick synopsis of the program data
  • Dashboard view gives status (i.e. number of results)
  • Several ad hoc reports are useful:
  • Assignment completion summary
  • Data overview
  • Results and reflections

Automate: Analysis of Student Data

  • All workbooks are stored in the TracDat document repository
  • Each semester’s data is in a separate folder
  • Download all the workbooks
  • An R script is used to process the workbooks
  • All data is aggregated into one csv file
  • Identifiable information is removed

Automate: Reporting

  • Using RMarkdown
  • Reuseable code
  • PDF reports for printing
  • HTML reports for distribution and collaboration

Promoting Assessment

  • True assessment requires faculty to evaluate and use the results!
  • Need to provide opportunities for interaction
  • Presentations in meetings
  • Moderated discussion boards
  • Comments in Open Science Framework
  • GitHub pull requests

Assessment in Action

Faculty Training

  • Frequent workshops of offered
  • One-on-one training sessions
  • Online instructional videos

Questions?

  • All of these resources are linked to from your handout
  • Licensed using either CC-BY 4.0 or MIT
  • Feel free to contact me if you have questions or comments